Field of the Invention
The present invention relates to a method for generating an augmented reality image for supporting the adjustment of the position of a surgical instrument during sinus surgery
Brief Description of the Related Art
There are several diseases known which can cause airway narrowing or stenosis. Inflammatory conditions belong to those known diseases. While there are several methods available to improve airway narrowing, balloon dilatation may provide a gentle, uniform dilation with less airway trauma than endoscopic sinus surgery with cutting instruments.
Physicians use during surgical planning a simplified planning scheme, wherein the cells or cavities of the nose are painted on paper to achieve better orientation in the complex anatomy of the sinuses. In such planning schemes simple three-dimensional geometric primitives are used to define the position and size of nasal cavities. Such a scheme supports the physician to transfer acquired generalized medical knowledge to the current situation in a patient and to detect the relevant anatomical structures and classify their geometric position. Such acute knowledge about the patient at hand is crucial to perform surgical interventions to achieve results within the medical state of the art.
In the literature, there are approaches to support the clinician by automatic and semi-automatic segmentation methods of the nasal anatomy during surgical planning. Here, the aim is to detect and identify the cavities of the nose in order to find or define the optimal access path to the surgical region of interest (ROI).
Zein and colleagues described 3-D region growing segmentation (Zein et al., 2005 Bildverarbeitung für die Medizin 2005 (pp. 93-97). The description discloses a contrast enhancement of 3-D image data and edge-preserving smooth filtering, a 3D region growing starting at seed voxels, which fulfil intensity threshold criterion followed by a post-processing of segmentation to reject areas not belonging to the nasal system by detecting leakage regions. The results are used for a 3D view of air space of paranasal sinuses and virtual endoscopy
Disadvantages related to the method disclosed by Zein et al are that the method only works with air-filled nasal cells. Further, it does not provide a differentiation and subdivision of individual nasal cells and the detection of natural drainage pathways of the sinuses.
Lo and de Bruijne described a voxel classification based airway tree segmentation (Lo, P., & de Bruijne, M. 2008, Proc. SPIE). They described a 3D region growing starting with main bronchus as seed point. They define with Kth nearest neighbor (KNN) a local criterion using the following image descriptors instead of using only image gray value: convolution with Gaussian, 1st and 2nd order derivates, gradient magnitude, Eigenvalues of Hessian, Laplacian, Gaussian curvature, Eigen magnitude, ratios of eigenvalues.
It is disadvantageous that the method disclosed by Lo and de Bruijne is only applicable for segments air-filled cavities and adjusted for use to segment airways in the lung and not suitable for the determination of natural drainage pathways of the sinuses.
Tingelhoff and colleagues disclose a comparison between the manual and semi-automatic segmentation of nasal cavity and paranasal sinuses from CT images (Tingelhoff et al., 2007, Proceedings of the 29th Annual International Conference of the IEEE EMBS (pp. 5505-5508). This document discloses a 3D region growing segmentation using AMIRA 4.1 software (Mercury Inc., now: Visage Imaging Inc.). It is disadvantageous that only segmentation of connected cavities is possible and a manual definition of seed points for region growing algorithm is necessary.
Moral and colleagues disclose a 3D region growing segmentation using AMIRA 4.1 software for planning of a path from nostrils to maxillary sinuses, sphenoidal sinuses, ethoidal sinuses and frontal sinuses (Moral et al. 2007 Proceedings of the 29th Annual International Conference of the IEEE EMBS, pp. 4683-4686). The disadvantages are same as mentioned above for the publication of Tingelhoff et al., namely that only segmentation of connected cavities is possible and a manual definition of seed points for region growing algorithm is necessary.
WO 2013/012492 A2 discloses a method for displaying a paranasal sinus region of a patient, comprising acquiring volume image data of the paranasal sinus region of the patient, identifying one or more airways within the paranasal sinus region from the volume image data, displaying the at least one or more airways and highlighting one or more portions of the displayed one or more airways that are constricted below a predetermined value.
WO 2013/012492 A2 relates to CT/Cone-Beam imaging, adaptive segmentation based on the disclosure of Pappas (Pappas, 1992, IEEE Transactions on Signal Processing, 40(4)), K-means classification (clustering) of segmented region based on their mean gray values (e.g. 4 clusters), voxel assignment to a cluster, external air removal using morphological operators on tissue maps/clusters and user interaction for modification of the result of the automatic segmentation, as can be taken from FIG. 4. This figure shows an added user input step 405 allowing the user to provide input that improves automatic segmentation, including edits to the class map, for example. In step 405, the user further provides instructions that modify the display of anatomy of the nasal region in some way. The modifications can be indicated interactively by viewer instructions entered with reference to a displayed rendering of CT slices in a coronal, axial, sagittal, or other view. The user input instructions can be entered using a pointer device, such as a computer mouse or joystick, for example, or using a touch screen as input device. Alternatively, the user may interact with the system using a 3D rendering of the nasal region. For example, in step 405 the user may enter instructions that indicate that an ostium of the left maxillary sinus is blocked. The indication that a ostium is blocked will cause a specific colour to display that cells may change. It is possible that the user removes sinus cells, bone and other tissue from the display, to skeletonize airways (compute medial lines within objects), compute cross-sectional views along the skeletons (perpendicular to path) and highlight important locations (e.g. locations of global or local minimum cross-sectional area that may occur at sinus ostia or locations at which a drainage path is restricted, or points with high curvature).
The method disclosed in WO 2013/012492 A2 allows a virtual endoscopic view, provides a path finding algorithm and the registration of segmentation to a labelled atlas based on statistical data so that the anatomy is identified. Alternatively manual anatomy identification or labelling by the user (using lists, audible prompt) is possible. An atlas with irregularities helps to identify different anatomical variations (e.g. agger nasi cell). Cell properties can be displayed (volume (natural and air), location, presence of polyps or infections) and simplified graphical representation of anatomical structures based on segmentation results can be generated. This document discloses a system to execute segmentation and display of paranasal cavities
A disadvantage of the method disclosed in WO 2013/012492 A2 is the difficulty for the user to prove the results of automatic segmentation and optimize it, especially to separate connected cells. The method does not allow to separate connected cells, a corresponding description is missing. In addition, there is no description how to segment blocked ostia.
WO 2009/120196A1 discloses a method of and a system for a 3D workstation for security and medical applications. A rendering method of volumetric data is described, including highlighting detected regions using the contour of the object on 2D displays and 3D stereoscopic displays. The contour colours are differently from rendering the volumetric data without highlighting. This document describes only the highlighting of a marked region in 2-D or 3-D views, but no description how the regions are detected is provided.
EP 1941449 B1 discloses a method for rendering a surface indicative of an interior of a colon. The method comprises using volumetric data (202) indicative of the interior anatomy of a human colon to render a surface (102) indicative of an interior of the colon. The method is characterized by the steps of identifying a region of the rendered surface, which is suspected to include residual stool, and highlighting the identified region (104) on an image of the rendered surface (102). The contour of the region of interest (ROI) is highlighted in virtual endoscopic view. This document discloses only a rendering method related to the colon.
WO 2008/021702 A2 discloses a method of quantifying a sinus condition of at least one sinus cavity of a patient. The method comprises the steps of generating an image of the patient, locating the at least one sinus cavity in the image and quantifying the sinus condition of the at least one sinus cavity based on the image, the automatic density-based detection and location of sinus cavities, a manual identification of sinus cavities by a technician and quantification of the amount of fluid or polys in a sinus cavity to determine the sinus condition or progress of sinus condition. WO 2008/021702 A2 does not provide the implementation of the manual or automatic segmentation of sinus cavities
In summary, the existing planning methods and systems provide some approaches for automatic segmentation of mainly air-filled sinuses cavities. The automatic segmentation methods, e.g. 3D region growing or 3D adaptive segmentation with k-means clustering, work well with air-filled sinus cavities. The challenge in the planning of sinus surgery, however, lies rather in a separation and identification of individual cells in particular under the absence of air.
In case of diseases such as inflammation of the sinuses or polyps, single or multiple cells are filled with tissue or fluid and the automatic segmentation methods are likely to fail due to marginal grey value differences of some cartilaginous cell walls and mucosa or pathological cell filling. Also, the identification and labelling of separated cavities is an unsolved problem especially in the case of pathological anatomy. Also the different quality of the 3D image data is often a problem for automatic methods, which require a high resolution of the volume image data and a normalized, or uniform grey scale density map in order to guarantee satisfying results. Especially image data from older devices and Cone beam computed tomography (CBCT) often do not meet these conditions.
Systems known from the state of the art do not provide support for a planning scheme based on patient-specific 3D image data of the human body, particularly of the paranasal sinuses. Therefore, up to date the planning scheme can only be manually performed on paper with the following disadvantages:                Spatial incorrectness        Error-prone scaling/size assignment        Used cuboids or cylinders describe the shape of the cells sometimes inadequately        Insufficient applicability to the surgical intervention        
Thus, there is a need for a computer-assisted method for fast and easy-to-use segmentation of the relevant nasal cells during the surgical planning according to simplified planning schemes. The results of the planning are intended to be in such a format that they also can be used and visualized intra-operatively by surgical assistance technology such as a surgical navigation system. For endoscopic or microscopic surgery, navigation systems with augmented-reality support allow the display of planning data superimposed in real-time endoscopic or microscopic camera images.
The perception of depth information of superimposed data in the augmented reality representation is usually also a problem, since the distance between the visualised object and the camera is often difficult to recognize. Another challenge is the intraoperative positioning and alignment of instruments relative to planning object, particularly using the real-time endoscopic or microscopic imaging.